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13.  ABSTRACT  (Maximum  200  words) 

This  paper  describes  a  method  of  assessing  military  command  decisionmaking  using  simulation  experiments  and  dynamic 
modeling,  along  with  some  typical  results.  Our  simulation  studies  have  been  carried  out  since  1 988  in  the  Research, 
Engineering,  and  Systems  Analysis  (RE8A)  facility  at  the  Naval  Ocean  Systems  Center  (now  Naval  Command,  Control  and 
Ocean  Surveillance  Center)  under  contract  with  Pacific  Science  and  Engineering,  Inc.  and  Sonalysts,  Inc.  RESA  is  a  facility  for 
presenting  large-scale,  event-driven  battle  simulations  with  realistic  scenarios.  High  fidelity  simulation  is  available  for  multiple 
platforms,  sensors,  and  weapons  under  a  broad  range  of  tactical  and  environmental  situations.  The  RESA  facility  has  been  used 
primarily  for  training  naval  battle  force  staffs  and  for  studies  of  advanced  sensor  and  weapons  concepts.  Our  interest  was  to  use 
the  RESA  facility  for  man-in-the-loop  studies  to  determine  the  cognitive  and  decision  processes  of  warfare  commanders  during 
battle.  Our  simulation  experiments  measured  the  overall  effectiveness  of  the  C3I  system  with  the  performance  of  the 
commanders  as  its  principal  determining  factor. 

Tiie  aim  of  the  research  program  is  to  improve  the  design  of  future  C31  systems  by  identifying  and  ameliorating  human 
information  processing  and  decisionmaking  problems  associated  with  critical  battle  decisions.  Initial  work  has  been  in  the 
anti-i.br  warfar  e  (AAW)  area.  Of  particular  concern  are: 

•  Identification  of  critical  information  used  for  battle  decisions; 

•  Specification  of  the  time  sensitivities  for  that  critical  information; 

•  Determination  of  the  effects  of  information  characteristics,  such  us  density,  consistency,  ambiguity,  conflict,  delay,  and 
degradation  on  the  decision  process;  and 

•  Assessment  of  the  role  of  individual  characteristics  and  cognitive  mechanisms,  such  as  initial  expectations,  and 
tactical  decisionmaking  style,  in  producing  variability  in  decision  behavior. 
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SITUATION  DESCRIPTION 

Overview  of  the  Research  Program.  This  paper  de¬ 
scribes  a  method  of  assessing  military  command  decision¬ 
making  using  simulation  experiments  and  dynamic  model¬ 
ing,  along  with  some  typical  results.  Our  simulation 
studies  have  been  carried  out  since  1988  in  the  Research, 
Engineering,  and  Systems  Analysis  (RESA)  facility  at  the 
Naval  Ocean  Systems  Center  (now  Naval  Command, 
Control  and  Ocean  Surveillance.  Center)  under  contract 
with  Pacific  Science  and  Engineering,  Inc.  and  Sonalysts, 
Inc.  RESA  is  a  facility  for  presenting  large-scale,  event- 
driven  battle  simulations  with  realistic  scenarios.  High 
fidelity  simulation  is  available  for  multiple  platforms, 
sensors,  and  weapons  under  a  broad  range  of  tactical  and 
environmental  situations.  The  RESA  facility  has  been 
used  primarily  for  training  naval  battle  force  staffs  and  for 
studies  of  advanced  sensor  and  weapons  concepts.  Our 
interest  was  to  use  the  RESA  facility  for  man-in-the-loop 
studies  to  determine  the  cognitive  and  decision  processes  of 
warfare  commanders  during  battle.  Our  simulation 
experiments  measured  the  overall  effectiveness  of  the  C3I 
system  with  the  performance  of  the  commanders  as  its 
principal  dctcimining  factor. 

The  aim  of  the  research  program  is  to  improve  the  design 
of  future  C3I  systems  by  identifying  and  ameliorating 
human  information  processing  and  decisionmaking 
problems  associated  with  critical  battle  decisions.  Initial 
work  has  been  in  the  anti-air  warfare  (AAW)  area.  Of 
particular  concern  are: 

•  Identification  of  critical  information  used  for  battle 
decisions; 

•  Specification  of  the  time  sensitivities  for  that  critical 
information; 

°  Determination  of  the  effects  of  information  charac¬ 
teristics,  such  as  density,  consistency,  ambiguity, 
conflict,  delay,  and  degradation  on  the  decision 
process;  and 

•  Assessment  of  die  role  of  individual  characteristics  and 
cognitive  mechanisms,  such  as  initial  expectations, 
and  tactical  decisionmaking  style,  in  producing 
variability  in  decision  behavior. 

The  Simulation  Experimental  Method.  The  tactical 
scenario  under  siudy  involved  die  air  defense  of  a  carrier 
battle  force  against  a  large  raid  of  enemy  bombers  carrying 
air-lo-surface  missiles.  The  battle  force  included  Aegis 
cruisers,  destroyers,  and  combat  support  ships.  The  princi¬ 
pal  weapon  employed  was  the  F-14D  fighter-interceptor 
armed  with  modem  air-to-air  missiles.  A  hot  war  situation 
was  assumed  diroughout.  The  simulation  runs  consisted  of 
two  hours  of  operational  time  compressed  into  50  minutes 


of  real  time.  Variations  of  the  scenarios  were  constructed 
to  test  differing  levels  of  the  experimental  variables. 
Detailed  intelligence  and  instructions  were  provided  to  the 
commanders.  Research  assistants  interacted  with  the  com¬ 
puter  system  so  that  the  commander  was  not  required  to 
learn  simulation  language  or  procedures.  In  order  to 
standardize  the  scenar  io  for  the  purpose  of  comparing  the 
performance  of  different  commanders,  enemy  actions  were 
entirely  prescripted  for  all  simulation  runs. 

The  Command  Information-Decision-Action  Pro¬ 
cess  During  AAW.  The  anti-air  warfare  (AAW)  com¬ 
mander’s  decision  environment  is  a  complex,  high  tempo 
world  based  on  a  stream  of  raw  and  processed  data  inputs 
from  a  wide  variety  of  sensors  and  intelligence  sources, 
with  supporting  analytic  and  communication  systems.  He 
is  assisted  by  a  team  of  subordinates  who  carry  out  various 
complex  processing  tasks  that  refine  and  cleanse  the  data 
and  help  interpret  it.  The  commander  must  integrate  key 
information  to  formulate  and  update  his  tactical  picture, 
relate  this  picture  to  anticipated  events  and  planned 
responses,  and  make  a  series  of  decisions  regarding  the 
appropriate  actions  to  respond  to  the  threat.  This  process  is 
a  dynamic  one,  continually  requiring  adjustments  as  the 
combat  scenario  unfolds. 

Command  decisions  during  battle  arc  made  within  a 
mission  and  organizational  context  of  command  relation 
ships,  responsibilities,  and  interdependencies.  Decisions 
are  often  short-fused.  Each  decision  is  bounded  by  a  limited 
time  period  for  response  (or  decision  window)  during 
which  action  must  be  taken  in  order  to  be  effective.  Earlier 
or  later  action  compromises  effectiveness,  putting  the 
commander  under  great  pressure  to  act  in  a  timely  fashion. 
However,  timely  decisions  may  require  acting  on 
incomplete  or  ambiguous  information.  The  decision  to  act 
initiates  a  complex,  often  time-consuming  implementa¬ 
tion  process  involving  the  coordinated  actions  of  many 
other  actors  and  associated  man-machine  systems.  As  his 
decisions  are  implemented,  the  commander  must  continue 
to  monitor  the  unfolding  situation  and  assess  the  effective¬ 
ness  of  his  actions  in  order  to  decide  on  follow-up  or  cor¬ 
rective  actions. 

The  decision  process  and  the  decision  environment  arc 
frequently  less  than  optimal.  The  information  flow  may  be 
sporadic  or  delayed.  Critical  data  sought  by  the  commander 
may  not  be  available  when  he  needs  it.  Varying  amounts 
of  spurious,  ambiguious,  or  conflicting  data  may  be 
delivered.  On  the  other  hand,  his  information  environment 
may  be  extremely  dense  due  to  the  high  tempo  of  threat 
actions  and  processing  quirks  during  battle.  It  is  a  chal¬ 
lenging  task  for  him  to  selectively  scan,  process,  integrate, 
and  interpret  incoming  data  in  relation  to  the  Battle  Force 
plans  and  mission.  Naval  officers  bring  to  the  battle 
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unii|in'  i  omhmaiion  .  ol  knowledge,  skills,  aiul  decision 
abilities.  so  that  vaiiahihly  among,  individuals  perloinimg 
(lie  command  decision  role  is  inevitable.  Commanders’ 
expectations  during  die  situation  assessment  process  may 
lai  dilate  or  distort  their  assessments.  lo'cn  among  equally 
experienced  warfare  commanders,  the  interpretation  of  die 
data  stream  can  result  in  different  tactical  pictures,  confid¬ 
ence  levels,  and  interpretations.  Information  characteristics, 
such  as  density,  consistency,  ambiguity,  and  delay  may 
influence  a  commander’s  decisions  in  subtle  ways. 

METHODS  FOR  ASSESSING  DECISION  PROCESSES 

Simulation  Experiments.  Simulation  experiments  are  a 
combination  of  simulation  technology  with  the  experi¬ 
mental  method.  Simulation  technology  uses  the  computer 
to  present  complex,  dynamically  unfolding  stimuli  in  a 
manner  which  i$  highly  appropriate  and  realistic  in 
mimicking  the  shipboard  Combat  Information  Center 
environment.  This  simulation  evokes  the  decisionmaker’s 
“natural”  information  processing  and  decisionmaking 
behaviors  while  allowing  the  researchers  to  control  a  large 
number  of  theoretically  or  empirically  less  relevant  factors. 
The  decisionmaker's  behavior  is  tutther  elicited  by  the 
apparently  normal  response  of  the  simulation  to  his 
actions.  Application  of  an  experimental  iserspective  leads 
to  creation  of  variations  of  the  standard  scenario,  to  manip¬ 
ulate  hypodtesized  causal  factors  determining  or  impinging 
on  the  decisionmaker’s  behavior.  The  standardized  scenar¬ 
ios  may  be  used  to  expose  a  sufficient  number  of  qualified 
decisionmakers  to  statistically  test  the  influence  of  the 
factors  under  examination.  Since  we  have  found  minimal 
learning  effects  in  our  battle  simulations,  decisionmakers 
may  be  exposed  to  a  systematically  programmed  sequence 
of  scenarios  to  obtain  repeated  measures  and  bolster  the 
number  of  data  points. 

The  simulation  technology  enables  us  to  measure  a  num¬ 
ber  of  process  as  well  as  outcome  variables.  In  RESA  we 
aie  able  to  collect  on-line  data  on  the  decision  process  and 
its  consequences,  such  as  orders,  object  positions,  and 
engagements.  In  addition,  we  can  make  audio  recordings  of 
participants  as  they  communicate  their  thought  processes 
or  conduct  official  communications.  Research  personnel 
also  record  situation  assessment  reports  and  post-trial 
debriefings.  These  verbalizations  can  be  subjected  to  cx'en- 
sivc  protocol  analysis  prior  to  statistical  evaluation.  We 
can  obtain  outcome  measures  (e.g.,  kills)  by  playing  out 
the  simulation  until  all  the  weapons  arc  released  and 
engagements  arc  completed.  We  have  devised  and  examined 
various  measures  of  performance  and  off  tivcncss,  cncom 
passing  the  individual  commander  and  the  battle  force. 

Simulation  experiments  provide  a  considerable  degree  ol 
control  over  salient  variables  during  simulated  battle 
decisionmaking  by  warfare  commanders.  However,  the 
limited  number  of  qualified  subjects  (warfare  area  experts) 
requires  limiting  each  study  to  a  few  variables  which  can 
be  systematically  manipulated  while  obtaining  adequate 
cell  sizes  for  statistical  analyses.  Analysis  of  a  larger  num¬ 
ber  ol  vaiiables  (and  levels  within  variables)  requires  that  a 


carefully  planned  sequence  ol  studies  he  perlnriued  I  Ins 
research  program  has  so  lar  Ionised  on  selected  minima 
lion  degradation  conditions  and  their  effects  on  the  deci¬ 
sionmaking  process.  A  jxissihle  future  locus  is  testing 
system  design  concepts  to  ameliorate  problems  identified 
in  our  studies. 

We  have  used  the  simulation  experimental  technology  to 
study  both  the  case  of  a  single  decision  maker  and  multiple 
interacting  warfare  commanders.  Studies  of  multiple 
commanders  entail  additional  cost  in  data  collection  and 
analysis  effort,  but  they  extend  the  range  of  questions  and 
issues  which  can  be  addressed.  For  example,  such  studies 
allow  examination  of  issues  in  information  exchange, 
methods  for  dealing  with  discrepancies  in  data  .interpreta¬ 
tion  among  commanders,  mechanisms  for  coordination, 
and  influence  patterns  among  commanders. 

Dynamic  Modelling.  Based  on  empirical  data  collected 
during  simulation  experiments,  a  model  of  warfare  com¬ 
mander  decision  processes  is  being  used  to  test  the  key 
decisions,  associated  variables  in  the  decision  process,  their 
causal  and  quantitative  relationships  to  each  other,  and 
their  relationships  to  system  (battle  force)  outcomes.  The 
model  has  two  levels  of  analysis  encompassing 
hypothetical  constructs  from  relevant  theory: 

•  The  macro  level,  in  which  the  interacting  decision¬ 
maker  carries  out  his  mission  responsibilities  within 
the  context  of  the  multi-person,  highly  structured, 
hierarchical  organization  of  the  naval  battle  group;  and 

•  The  micro  level,  in  which  the  warfare  commander 
functions  as  the  decisionmaker  responsible  for  pro¬ 
cessing  information  and  solving  problems  in  order  to 
arrive  at  action  decisions  in  his  warfare  area. 

The  model  uses  quantitative  data  from  the  simulation 
experiments,  fleet  exercise  evaluations,  and  other  empirical 
studies  to  precisely  define  the  relationships  among  key 
factors  operative  in  the  command  decisionmaking  process. 
The  model  helps  us  understand  hov/  data  flows,  time 
phasing,  interpretation  mechanisms,  problem  solving 
processes,  cognitive  factors,  and  interpersonal  factors 
affecting  the  decision  processes  of  warfare  commanders. 
The  empirically  defined  relationships  among  factors  in  the 
model  provide  an  unambiguous  framework  for  deriving 
testable  research  questions,  hypotheses,  design  principles, 
and  guidelines  for  improving  C3I  support  for  the  decision 
process.  A  select  few  of  these  derivations  will  be  further 
tested  in  inan-in-the-loop  studies.  Thus,  use  of  the  model 
helps  guide  studies  of  the  decision  process. 

TYPES  OE  RESULTS 

Use  of  the  simulation  experimental  method  and  dynamic 
modelling  has  produced  several  insights  into  the  command 
decision  process. 

Process  Analysis.  By  performing  protocol  analysis  ol  the 
commanders’  concurrent  verbalizations  about  their  decision 
processes  during  die  simulation  experiments,  we  have  been 
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able  to  track  then  use  ol  battle  information  in  making 
launch  decision:;.  A  clear  pattern  in  information  usage 
emerges  when  the  battle  is  divided  into  pluses  defined  by 
scenario  even’s,  as  shown  in  Figure  I.  Considering  only 
threat  information,  Raid  Count  was  used  most  across  all 
battle  phases,  peaking  in  the  Late  phase.  Use  of  Intelli¬ 
gence  Reports  and  Track  Localization  information  varied 
most  across  battle  phases,  both  showing  a  pattern  of  ini¬ 
tially  high  usage  that  decreased  steeply  as  the  battle 
progressed.  Electronic  Supi>ort  Measures  (ESM)  and  'brack 
Speed  information  were  used  least  often,  with  ESM  being 
most  frequently  used  early  in  the  battle  and  Track  Sjxxxl 


Intel  Rpti  ESM  Track  Track  Spead  Raid  Count 

Locatljutlon 


Threat  Information 


Figure  1.  Use  of  threat  information  across  battle  phases 

late  in  (Jie  battle.  From  the  standpoint  of  optimal  decision 
making,  die  observed  pattern  of  information  usage  drasti¬ 
cally  underemphasizes  intelligence  information  early  in  tire 
battle,  ESM  throughout  the  battle,  and  track  information 
in  the  middle  and  end  phases  of  die  battle.  It  appears  that 
graphically-presented  data  is  overemphasized  and  abstract 
alphanumeric  data  is  underemphasized  or  poorly  integrated 
into  the  tactical  picture.  Our  analysis  suggests  the  possi¬ 


bility  that  improved  presentation  ol  relevant  tactical  data 
could  be  used  to  change  the,  warfare  commander's  pattern  ol 
usage  to  achieve  improve  battle  outcomes. 


(ienerai  Factors.  Ou.  locus  has  been  on  measuring  the 
influence  of  factors  that  exert  strong  general  effects  on  the 
decision  process  of  the  typical  commander.  We  have  found 
diat  short  interruptions  in  the  How  of  tactical  information 
have  highly  consequential  disruptive  effects  on  launch  deci¬ 
sions,  although  commanders  have  little  awareness  of  the 
alteration  in  their  behavior  or  its  ultimate  impact  on  sys¬ 
tem  performance.  (Sec  Figure  2.)  In  examining  decisions 
to  launch  interceptors,  interrupts  during  the  early  phase  of 
die  batdc  (before  the  main  body  of  the  enemy  was  detected) 
were  found  to  significantly  affect  outcomes  (i.c.,  kills). 
Since  commanders  were  waiting  for  positive  indications  of 
the  location  of  die  main  raid,  greater  interruption  caused 
them  to  delay  their  decisions  to  launch  die  bulk  of  their 
assets  until  definite  raid  indications  were  seen.  Those 
delays  were  critical  to  system  outcomes,  because  they 
reduced  the  possibility  of  successful  completion  of  the 
intercepts.  Although  the  number  of  launches  were  equiv¬ 
alent  in  the  middle  and  late  phases,  the  effects  of  the  delays 
in  the  early  phase  were  not  able  to  be  offset  by  later 
launches  due  to  the  geometries  and  rime  requirements. 

Individual  Factors.  Initial  studies  suggested  apparent 
stylistic  differences  in  decisionmaking  that  we  have  defined 
as  spontaneous  vs.  deliberadve.  Spontaneous  commanders 
rely  more  on  die  emerging  graphical  tactical  picture,  widi 
less  anticipatory  behavior  and  using  fewer  kinds  of  supple¬ 
mentary  information,  such  as  ESM,  in  making  launch 
decisions.  Their  launch  decisions  result  in  relatively  delay¬ 
ed  resource  commitment  and  more  assets  left  unused  on 
deck.  Deliberative  commanders  use  more  kinds  of  infor¬ 
mation  and  display  a  more  anticipatory  orientation,  com 
milting  their  resources  earlier  and  employing  more  of  dicir 
available  resources  against  the  threat.  (Sec  Figure  3.) 
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Figure  2.  Anti-air  warfare  commander  lag  in  early  resource  commitment  under  tactical  data  interrupt 
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Figure  3.  Influence  of  decision  style  on  launch  profile 
CONCLUSIONS 

Command  decisionmaking  is  an  extremely  challenging  set 
of  phenomena  to  study.  In  its  natural  setting,  decision¬ 
making  is  a  complex,  one-time  event.  It  is  performed  by 
relatively  few  people  whose  extensive  experience  and 
knowledge  are  bases  for  rare  expertise.  It  is  largely  an 
internal  process  whose  most  important  mechanisms  arc 
probably  not  appreciated  by  those  who  do  it  best.  It  is 
likely  influenced  by  factors  from  many  domains,  ranging 
from  task  and  information  characteristics  to  organizational 
structures  and  cognitive  processing  mechanisms.  To  make 
these  phenomena  tractable  requires  a  method  that  can 
enable  quantification  in  the  laboratory  without  losing  its 
essential  nature. 

A  critical  aspect  of  the  nature  of  command  decisionmaking 
is  the  dynamic  information  environment.  To  attenuate  the 
information  flows  would  fundamentally  change  the  nature 
of  any  decisionmaking.  To  eliminate  or  minimize  the 
experience  and  knowledge  brought  to  the  task  by  its  practi¬ 
tioners  would  alter  the  phenomena  in  unknown  ways.  To 
reduce  the  stress  or  tempo  or  ambiguity  or  uncertainty 
would  call  into  question  the  validity  and  gcncralizability  of 
any  findings.  The  simulation  method  is  essential  to  main¬ 
taining  these  critical  aspects  of  decisionmaking 
phenomena. 

Systematic  analysis  requires  careful  manipulation  of  fac¬ 
tors  hypothesized  to  have  an  influential  role.  The  com¬ 
plexity  and  subtlety  of  the  impacts  and  interactions  among 
these  factors  dictate  a  high  degree  of  control,  using  simula¬ 
tion  experimental  designs  that  hold  most  factors  constant 
while  varying  a  few.  The  most  important  factors  can  be 
explorer!  using  dynamic  modelling  to  test  additional  levels 
and  combinations.  Again,  the  simulation  technology  allied 
with  an  experimental  paradigm  and  modelling  tools 
provides  a  workable  approach. 


static  approaches  led  to  a  conceptualization  ol  decision 
making  as  occurring  at  a  single  point  in  lime.  The  sun 
illation  experimental  method  allows  a  fundamentally  dil 
ferent  picture  of  tactical  decisionmaking  to  emerge  than 
has  been  understixxl  previously.  Tactical  command  deci 
sionmaking  should  no  longer  be  viewed  as  a  momentary 
calculation  of  expectancies  or  comparison  of  options. 
Rather  it  should  lx;  understood  as  beginning  with  the  tacti¬ 
cal  planning  process,  when  anticipated  events  are  defined 
and  response  strategics  evaluated  and  laid  out  along  a  time¬ 
line  in  coordination  with  other  interdependent  members  of 
the  command  team.  A  later  phase  of  the  same  decision 
process  is  the  assessment  of  the  unfolding  situation  in 
relation  to  the  previously  formulated  plans.  This  assess¬ 
ment  may  require  weighing  the  appropriateness  of  execut¬ 
ing  planned  responses  as  opposed  to  modifying  the  plan¬ 
ned  responses  to  suit  circumstances  which  deviate  from 
those  anticipated  during  planning.  The  decision  process 
continues  beyond  the  “decision  point,”  into  the  execution 
process,  when  feedback  may  dictate  decisions  to  take  cor¬ 
rective  or  additional  actions.  In  essence,  the  entire  defini¬ 
tion  of  decisionmaking  should  be  shifted  from  a  moment¬ 
ary  event  to  a  process  over  time.  This  opens  new  vistas  for 
understanding  the  influential  factors  and  mechanisms  and 
how  such  factors  can  be  controlled  or  brought  into  the 
service  of  improved  decisionmaking. 

Command  decisionmaking  is  an  integral  component  of 
weapon  systems  performance.  That  decisionmaking  is 
based  on  C3I  systems  that  provide  supporting  information 
and  assistance  in  processing  iL  There  is  a  growing  realiza¬ 
tion  among  military  systems  designers  that  a  critical 
amount  of  the  performance  anticipated  during  design  fails 
to  occur  when  systems  are  fielded  due  to  inadequate  consid¬ 
eration  of  the  users’  needs  and  capabilities.  Our  findings 
show  that  commanders  do  reasonably  well  using  a  portion 
of  the  available  data  (primarily  radar),  but  other  data  (i.c. , 
track  speed,  ESM)  is  not  integrated  and  used  effectively. 
The  causes  of  this  distortion  in  information  use  may  be 
organizational  as  well  as  cognitive  and  must  be  further 
investigated.  Modem  thinking  recognizes  the  human  user 
as  an  integral  component  of  the  system  and  a  key  determi¬ 
nant  of  its  |)crformance.  Efforts  to  improve  the  design  of 
future  C3I  systems  require  better  understanding  of  the 
systemic  causes  of  variability  and  degraded  human  decision 
performance.  The  simulation  experimental  methodology 
and  dynamic  modelling  tools  provide  a  much  needed 
capability  to  measure  complex  decision  behavior  relevant 
in  military  'pcrational  settings.  The  NCCOSC  research 
program  is  plying  these  methods  as  tools  for  producing 
insights  ami  testing  solutions  that  can  be  used  to  improve 
decision  effectiveness  in  critical  operational  situations. 


!  Presented  at  the  9th  Annual  Decision  Aids  Working 
Group  Conference,  Monterey,  CA,  8-9  June  1992. 

^  This  work  was  supported  by  Office  of  Naval  Techno¬ 
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Any  research  method  imposes  constraints  on  the  concep¬ 
tual  understanding  of  the  behavior  under  study.  Previous 
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